Systems and methods for byte-level or quasi byte-level single instancing

ABSTRACT

Described in detail herein are systems and methods for deduplicating data using byte-level or quasi byte-level techniques. In some embodiments, a file is divided into multiple blocks. A block includes multiple bytes. Multiple rolling hashes of the file are generated. For each byte in the file, a searchable data structure is accessed to determine if the data structure already includes an entry matching a hash of a minimum sequence length. If so, this indicates that the corresponding bytes are already stored. If one or more bytes in the file are already stored, then the one or more bytes in the file are replaced with a reference to the already stored bytes. The systems and methods described herein may be used for file systems, databases, storing backup data, or any other use case where it may be useful to reduce the amount of data being stored.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. PatentApplication No. 61/118,379, filed on Nov. 26, 2008, entitled SYSTEMS ANDMETHODS FOR BYTE-LEVEL OR QUASI BYTE-LEVEL SINGLE INSTANCING, which isincorporated by reference in its entirety.

BACKGROUND

Conventional single instancing solutions are typically file-based orblock-based.

File-based means that, for example if an Excel spread sheet file on adisk is copied to another location on the disk, on a single instanceddata store, it would be detected that they contain the same contents andonly one reference to the file would be kept regardless of how manytimes it appears on the disk (or other disks) and under whichfilesystems.

However, if this Excel spreadsheet is opened and only one number ischanged then it is saved, even though 99% of the document is still thesame, file-level single instancing would not work.

Block-level single instancing would help here. Block-level singleinstancing chops up files into blocks and even if one block (in theexample where the number got changed) doesn't match, the other blocksstill do, so they can be single instanced.

However, if new rows are inserted, or using a different example, a filee.g., a Word document, that is split into blocks of certain fixed blocksizes, if somewhere at the start a single letter, a word, or a sentenceis inserted, everything below it will shift and all the blocks willbecome different even though only a small change in the file occurred.

The need exists for a system that overcomes the above problems, as wellas one that provides additional benefits. Overall, the examples hereinof some prior or related systems and their associated limitations areintended to be illustrative and not exclusive. Other limitations ofexisting or prior systems will become apparent to those of skill in theart upon reading the following Detailed Description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example environment in whichaspects of the invention may be configured to operate.

FIG. 2 is a block diagram illustrating components of a data stream thatmay be used in a suitable data storage system.

FIG. 3 is a block diagram illustrating an example of a suitable datastorage system.

FIG. 4 is block diagram illustrating an example of components of acomputing device used in data storage operations.

FIG. 5 is a flow diagram illustrating an example process for determiningif a code block is equal to other code blocks.

FIG. 6 is a flow diagram illustrating an example process for populatinga data structure with code blocks.

DETAILED DESCRIPTION

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of the claimed invention.

Single Instancing and Overview

One purpose of single instancing (alternatively referred to herein asdeduplicating) is to minimize storage space and to avoid storingduplicate data multiple times.

One challenge of performing single instancing on a byte-level is findingmatches. With a block-level where a strong hash function is used, eitherthe whole block can be used, or nothing at all. Also the strong hashfunction (such as MD5 or SHA1) may present a problem in that a sequencelike ‘1234567890’ and a hash of length 4, to calculate the hash of‘1234’, ‘2345’, ‘3456’, ‘4567’, ‘5678’, ‘6789’, ‘7890’ requires eachbyte (7*4=28 bytes here) to be read again and again.

Using a rolling hash reduces the number of reads to ‘1234’, −1+5, −2+6,−3+7, −4+8, −5+9, −6+0; or to 16. Rolling hashes reduce hash calculationtime from one byte to the next which is needed in byte-level singleinstancing from O(n) to O(1). Even when a hash is present, a hash onlyallows one to verify in one shot if a set number of bytes might be equalto another set number of bytes. It breaks down if the number of bytesone wants to compare becomes variable.

In addition, when wanting to compare whether or not data of a variablesize matches anything already encountered, how to do the comparison a)the size of the data can be very small or extremely large and b) it maybe difficult to ascertain what the size of the data to compare is.

This is where the ‘power of 2’ hashes can be used. By that, a number ofhashes is stored rather than just one, where the amount of data hashedincreases by powers of 2. Rolling hashes may not be reliable though, sothe data needs to be verified. If one would just link to the data storedin different files, one would need to open and close hundreds andhundreds of files every minute. This may not work very well.

This document describes systems and methods that provide for storingdata in a manner employing byte-level or quasi byte-level singleinstancing so that individual files or data objects can be read in arandom access fashion, modified and space can be reclaimed for reuse.

In some embodiments, file names or other metadata of files may bestored. In other embodiments, such metadata may not be stored. In theseembodiments, however, the data of a file can be referenced by way of areference pointer.

This makes the system useful not only for filesystems, but anywherewhere data needs to be referenced and it is desired to reduce the amountof data storage. This approach would also be useful for storing blobdata or large strings in databases, or as in a use case of storingbackup data.

The system will store data in blocks, however not the whole block needsto match for single instancing; e.g., only the start of the sequence ofdata stored within a block needs to match in order for the block tobecome usable. Block sizes of data can be very small. The smaller theblock size, the better the single instancing achieved, however also thehigher the overhead. Blocks can reference other blocks, by way of directaddressing and a length (e.g. block 10 can reference block 2 starting atoffset 26 for a length of 200 bytes source bytes). Due to thisreferencing, even though the block size is very small, one block canrefer to gigabytes of actual data (from now on referred to as “sourcedata”). One implementation can also have pseudo blocks which are notcoded data blocks, but rather links to inside other coded data blocks.This may be useful to put a frequently used offset of an actual blockinto the search tree. A byte-level or quasi byte-level single instancingmethod allows single instancing of a block even though the entire blockdoes not match another block (only just the beginning, or with areference even only a part within it). This differs from block- orfile-level single instancing where if data shifts by a byte, a lot (ifnot all) blocks become out of alignment and cannot be used. Withbyte-level or quasi byte-level single instancing, the maximum amount ofdata which cannot be single instanced due to such a data shift is oneblock length (in the byte rather than kilobyte range). Blocks mayfurther be stored within a search tree. Pseudo blocks may be insertedinto the search tree, but do not have to be. The search tree allows forrandom access of desired blocks.

This search tree should allow direct addressing of the nodes—so that onenode or leaf of the search tree can immediately link to another node orleaf of the search tree, no matter where in the search tree thereferenced block is. The resulting data structure may be considered tobe a graph. This data structure will be referred to herein as the“search tree.” In some embodiments, a node should not refer to any nodethat was added after the node so as to avoid cyclic dependencies.

The search tree portion will contain coded data blocks in a sortedfashion so it is easily traversable. Each block contains metainformation—for example but not limited to: the number of references,which byte within the block contains the half way mark of the data andwhat the source offset is, and hashes.

In some embodiments, the data is to be stored in very few files (maybeonly one file). This helps during verification that is required, andadds the benefit that any number of bytes from a sequence alreadyexisting can be used—also this adds the advantage of being able toupdate the reference counts of the individual code blocks whiletraversing the data. In addition, storing the data in this fashioncompresses the data on the fly.

The data should be stored in a sorted format—so, in some embodiments,that data is stored in a search tree. Any suitable data structure may beused. The data structure should be a data structure where any node ofthe search tree can reference other nodes of the search tree regardlessof where in the search tree the other nodes are, possibly subject tosome exceptions. In some embodiments, a first node of the search treecan reference data in any other node excepting nodes that aredescendants (e.g., children, grandchildren, etc.) of the first node.

A suitable mechanism of how to compare and sort two data blocks that canrepresent data gigabytes in length (though the actual blocks are only acouple of bytes long) is needed.

In some embodiments, performing byte-level or quasi byte-level singleinstancing uses the following method: by using power of 2 rolling hashesto sort a large number of small blocks of data representing a very largeamount of data into a search tree where each node can reference data inother nodes, for the purpose of storing data in a compressed and singleinstanced format, whereby it is still possible to easily jump to anypoint within a file and read from it, and also being able to modify anddelete data from this data structure.

The advantages of byte-level or quasi byte-level single instancing arenumerous: a lot more data than previously with block-level or file-levelcan be single instanced and an automatic compression is achieved at thesame time. Since hashes are in general not relied upon (they may be usedto detect repeat occurrences) there are generally also no problems ofhash collisions—which could cause data corruption in previousapproaches.

Various examples of the invention will now be described. The followingdescription provides specific details for a thorough understanding andenabling description of these examples. One skilled in the relevant artwill understand, however, that the invention may be practiced withoutmany of these details. Likewise, one skilled in the relevant art willalso understand that the invention may include many other obviousfeatures not described in detail herein. Additionally, some well-knownstructures or functions may not be shown or described in detail below,so as to avoid unnecessarily obscuring the relevant description.

The terminology used below is to be interpreted in its broadestreasonable manner, even though it is being used in conjunction with adetailed description of certain specific examples of the invention.Indeed, certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this Detailed Descriptionsection.

The following discussion provides a brief, general description of asuitable specialized environment in which the invention can beimplemented. Those skilled in the relevant art will appreciate thataspects of the invention can be practiced with other communications,data processing, or computer system configurations, including: Internetappliances, hand-held devices (including personal digital assistants(PDAs)), wearable computers, all manner of cellular phones, mobilephones, and/or mobile devices, multi-processor systems,microprocessor-based or programmable consumer electronics, set-topboxes, network PCs, mini-computers, mainframe computers, and the like.The terms “computer,” “server,” “host,” “host system,” and the like aregenerally used interchangeably herein, and refer to any of the abovedevices and systems, as well as any data processor.

While aspects of the invention, such as certain functions, are describedas being performed exclusively on a single device, the invention canalso be practiced in distributed environments where functions or modulesare shared among disparate processing devices, which are linked througha communications network, such as a Local Area Network (LAN), Wide AreaNetwork (WAN), and/or the Internet. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

Aspects of the invention may be stored or distributed on tangiblecomputer-readable media, including magnetically or optically readablecomputer discs, hard-wired or preprogrammed chips (e.g., EEPROMsemiconductor chips), nanotechnology memory, biological memory, or otherdata storage media. Alternatively, computer implemented instructions,data structures, screen displays, and other data under aspects of theinvention may be distributed over the Internet or over other networks(including wireless networks), on a propagated signal on a propagationmedium (e.g., an electromagnetic wave(s), a sound wave, etc.) over aperiod of time, or they may be provided on any analog or digital network(packet switched, circuit switched, or other scheme).

Additional Features

With block-level and file-level single instancing, there is only verylittle data (the strong hash) to check against a hash database, onelookup every 128 KB or more—not looking for a match for every singlebyte read. How to do this fast and quickly determining the best match isone problem.

Databases store data within trees also; however they store the completerow data. In some embodiments, the data is being divided up into tinypieces, compressing it into the tree and making all the partssearchable. While hashes are used—they are not relied on, they are justused to find potential matches.

Suitable Data Structures

The search tree should have the following capabilities: 1) as balancedas possible with minimal overhead when adding new code blocks; and 2)while the positions of blocks within the search tree may change, theblock should be addressable under its old location even if the locationin the search tree changes due to inserts/deletions.

In some embodiments, the data structure is “skipped lists.” A skip listis a probabilistic data structure, based on multiple parallel, sortedlinked lists, with efficiency comparable to a binary search tree (orderlog n average time for most operations). However, another datastructure, for example a “red-black tree” linking to coded blocks, maybe used.

One advantage of skipped lists is that it remains mostly balanced, doesnot require reordering of nodes when items are inserted/removed (meaningabsolute locations will remain the same), and the sorted linked listproperty of it also lends itself well to objectives of byte-level orquasi byte-level single instancing. It also makes locking of nodeseasier and keeps the rest of the search tree open in case of concurrentaccesses. An entire block can be bypassed by providing alternate pathsduring locking and still have a valid search tree.

An important consideration is how to store the data within the searchtree and how to determine if one block (which can refer to gigabytes ofsource data) is smaller, equal or greater than an existing block.

In some embodiments, the method used is rolling hashes. A rolling hashis a hash function where the input is hashed in a window that movesthrough the input. In some embodiments, Robin-Karp rolling hashtechniques may be used.

Rolling hashes have the property that given an input stream and a fixednumber of n bytes for which a hash is to be created, that it istypically very easy and fast to get the next hash when the next inputcharacter is read (example: sequence is ‘1234567’ a hash of 4 bytes istaken, so hashes of ‘1234’, ‘2345’, ‘3456’, ‘4567’ need to becalculated). With other hash mechanisms to calculate a new hash, thehash needs to be calculated using all n bytes, for a rolling hash, thehash is only dependent on the previous hash and the character droppingout and the character coming in, which may lead to speed advantages overother techniques.

In some embodiments, there is not one rolling hash, but several—it isthe ‘hashes of the powers of 2’—a hash of the first 64 source bytes, thehash of the first 128 source bytes, a hash of the first 256 sourcebytes, etc.

Since there is a certain minimum number of bytes a sequence of bytesshould have in order to be worthwhile to single instance, the hash ofthis minimum sequence length would be an additional hash.

It may have advantages if this minimum hash is itself a power of two(say 32 bytes length=2̂5). Anything below this may not be singleinstanceable. Not all ‘power of 2’ hashes for a block will be stored.The minimum sequence hash is stored and is the first sort criteria.However, the next power of two stored for a block could be determined bythe logarithm of the amount of source bytes stored in the first half ofthe block, though other optimizations could also be applied.

In some embodiments, not all blocks are inserted into a search tree. Ifa code block fills less than half the amount of space, it will get astorage location, but may not be included into the search tree. Thisalgorithm could be varied to %, or the whole or any other fractionwithout changing the invention.

In regards to the sort order in a tree: Whether one code block is lessthan another can be determined using an example process illustrated inFIG. 5. The process 500 begins at step 505 where a code block isreceived. At step 510 the minimum sequence length rolling hash of sourcebytes is compared to others in the search tree. If it is not found, thenthe process 500 continues to step 530, where it is determined that thecode block is not equal to other code blocks. If the minimum sequencelength rolling hash is found, the process continues to step 515. Foroverlapping power of 2 hashes, in order of lowest power of 2 hashes, acheck for equality is made. If any is less or greater than the other,then the process 500 continues to step 530, where it is determined thatthe code block is not equal to other code blocks. Otherwise, the process500 continues to step 520. If all are equal or no overlapping power of2's found, then base the decision on the length of the amount of sourcebytes. If they are also equal, then the process continues to step 525,where the blocks are considered equal. If they are not equal, then theprocess 500 continues to step 530, where it is determined that the codeblock is not equal to other code blocks. The process 500 then ends.

Slight variations (such as taking into account other criteria such asthe source length of the half way mark in the block etc) are alsopossible.

Note that in the above, there are greater degrees of being not equal.Two blocks can totally mismatch, or they can match partially. The morepower of 2 hashes that match, the better the match, even if they do notall match.

Data Storage

The following types of data should be stored on a physical data storagemedium (e.g., a computer-readable storage medium): 1) search tree skiplist base pointer list; 2) actual code blocks themselves; 3) pseudo codeblock containing a link inside a code block; and 4) skip list node block(pointer to previous and next skip list node, pointer to code block).

These different entities may be stored in various ways. What should bestored is what they contain and which code blocks refer to which othercode blocks, and not necessarily how the data inside each block isstored. Each or any of the above may be stored separately. The codeblock should contain a number of code bytes going up to a small setmaximum.

A code byte can either be a literal byte, or it can be a reference toanother block. If it is a reference, the following data should be storedalso: 1) block number (large number as there can be a large number ofblocks); 2) offset inside block (this is small, will be a byte or less);3) number of code bytes followed inside block (will be a byte or less);4) number of source bytes contained (can be a large number); and 5)rolling hash of source bytes followed (usually 32 bits).

If, for example, 6 bytes are taken for the block number, 6 bytes for thenumber of source bytes followed, 4 bytes for the hash, and a byte eachfor codebyte offset and number of code bytes followed in block, thenthere are 18 bytes for the code byte—meaning that the sequence followedshould be longer than 18 bytes source bytes.

The literal code bytes would need to be at least a bit extra in length(9 vs. 8 bits) in order to signify that they are in fact literals.

A pseudo block would contain the pointer to the other block, an offsetmeasured in code bytes, an offset measured in source bytes and adifferential rolling hash (which with modulo exponentiation can be usedto calculate the power of 2 hashes of the pseudo block using thelinked-to block's power of 2 hashes). Modular exponentiation is a typeof exponentiation performed over a modulus.

The power of 2 hashes need not necessarily be stored, only onedifferential. This makes it light weight to store pseudo blocks in thesearch tree.

Populating the Search Tree

FIG. 6 is a flow diagram illustrating an example process for populatinga data structure with code blocks. The process begins at step 605, wherean input stream of data is received. Given the input stream, at step610, the hashes of the power of 2s are generated. This is an operationwhich generally requires minimal computational complexity since only thefirst and last bytes need to be added and/or removed.

At step 615, for each byte, it is looked up whether there exists anentry in the search tree matching the hash of the minimum sequence. Atstep 620, it is determined whether a match is found in the datastructure. If found, then the comparison as mentioned earlier alsofeatures comparing the power of two hashes, the one matching the mostnumber of power of 2 hashes is evaluated first.

By evaluating, what is meant is that matching code block is evaluatedand expanded, code byte by code byte, and compared against the inputstream. If the code byte is a reference, then the complete referenceshould match.

In some embodiments, the reference could be followed to a deeper level,for example, to a certain maximum level or to a maximum number of bytes.However in some embodiments, it was chosen not to go partially into areference, either a reference has to hold and be equal exactly or theother choice is to use a code block until the previous code byte in thecode block.

This is where the reference hash is used—by using modulo exponentiation,it is possible to reconstruct the hash of the code block up to the pointdirectly after the reference and compare it to the rolling hash whichwas built while following the tree. Only if after a reference the hashstill matches, is the link followed recursively and validated that itactually matches.

Even if there is a hash collision on a higher level, one level down isalready usually the place where a hash of another link will no longermatch.

The task of following the search tree can be split up into multiplethreads where one thread starts looking up the input stream at location0, the next thread at location 1, etc. If no match is found, it can goforward to an unprocessed location. If a match ends after less than acertain number of source bytes (for example, 18 or more, depending onvarious factors), then the thread can abort and also start from thebeginning at an unprocessed location.

For threads, when they have a match greater than a certain amount ofthreshold source bytes (could be larger than the minimum), it could alsoreset the unprocessed location to that number, so that any threads notfinding a match can start from that location. While that may not yieldthe optimal match, it may result in a match that is good enough.

If while going through a search tree, there are multiple candidates tofollow, then it can be up to the implementation to decide whether thefirst tried path was good enough. If from the matching hashes it isexpected that a candidate (perhaps to a hash collision) could be abetter match than the one found, then it can be tried again.

As soon as a path is determined to be sufficient, all the blockreference counters at this stage should be increased by one, also thereferences that are used following this.

At step 620, if no reference is found, then the literal byte is added tothe current block. The process then continues to step 635. At step 635,if a reference is found, then one is added to the reference count of theblock being referenced.

At step 635, it is determined if the bock is full. If not, the process600 returns to step 605. Once the block is full, at step 640, the blockis marked to be added to the search tree. The process 600 thenconcludes.

Second Level Pass

If a block that is marked to be added to a search tree contains multiplereferences in a row, and the total length of these references exceedsthe block size of the incoming data, then it should be checked if thereis not a block existing, which has all of the same references (oneoption is to first search for a block having matching hashes).

In some embodiments, the second level pass is performed as anoptimization. If no replacement can be done on a block, then it isfinalized and written to the search tree. The candidates for a secondlevel pass could be marked while scanning if it is found that a largenumber of the power of 2's match, but it is not yet possible to matchthe remaining bytes. In such a case the potential match could be addedto a buffer in memory and later checked.

Links to within Code Blocks

While processing the input buffer, an LZW algorithm could be run on thedata—not so much for actual compression of the data; but to findrepetitions in the input stream. LZW is a lossless data compressionalgorithm.

Normally it is only possible to refer to the beginning of a code block;however with this mechanism it is possible to link inside of a codeblock starting at, for example, the 15th code byte rather than thefirst.

For a current input stream, it is known which blocks were created by thecurrent input stream and so links inside those blocks can easily beconstructed; if it is determined that a link is “useful enough,” itcould also be added to a search tree as a pseudo block.

“Useful enough” could be defined by saving a significant percentage ofthe input stream in that link. Note that the links themselves generallywill not be referenced; only what the links link to will be referencedby other code blocks.

Output Data

The output will only be one reference to a block. All blocks that gotadded will be recorded in a separate block. If that block becomes full,it will itself be added as a block and a new block will be started whichcontains that block. One objective is that for each file stored, thatthere is one block. If a block is not full, it will only be stored butnot linked to in the search tree.

An Example

Take the following string: “Present your card with every order andreceive promotional discounts.” An idea is to build a search tree andstore the single instanceable data inside the search tree.

For sake of clarity, to describe the algorithm only blocks of 8 codewords will be used. In some embodiments, blocks of 64, 128, 256 bytes orperhaps even higher may be used.

Given the string above, the following code blocks are generated. The topvalue (01 02) is just some file offset. The value “(sp)” means spacecharacter. Given an empty tree, there is not much that can bereferenced, so the first few blocks look like the following:

All these are added to the search tree (sorted by the first fourcharacters and then the power of 2 hashes and the length). The nextblock is able to make a reference if the LZW extension described aboveis used. This is possible because blocks 1,10,17 and 25 come from thisinstance for which an LZW table has been created.

The next block (33) would look something like: (blocks are named in thisexample after their storage location in this example, it could have beenjust as easy to use B1, B2, B3, B4, etc.)

The syntax for (−>Bx, y, z) is: Reference to block x, offset y (startingat 0) and use z characters.

These references could be added to the search tree also by themselves.Certain criteria should be met, especially having been used in the past,and perhaps having been used a certain number of times or leading to acertain amount of data, or rank these the used links by a points systemand for every file added, add the top x.

In this example, links used will be added as pseudo blocks.

So block 41 will be (−>B17,1) (no length, just the start offset), block42 will be (−>B02,1), block 43 would be (−B25,1).

Now an unfortunate case for block-level single instancing would be ifthis string: “Present your card with every order and receive promotionaldiscounts.”—was changed to—“resent your card with every order andreceive promotional discounts.” In other words, the first letter wasdeleted.

Here, for the first character there is already a link already going to(B02,1). So it is written (−>B02, 1, 7) (7 characters), followed by thecomplete blocks B10,B17,B25 But even if this did not exist, B10, B17,B25 would have been used in any case. The seven characters would haveneeded to be written into a new block before the first reference.

Random Access

Each file links to one block. Each block has a half way marker and avalue saying how many source bytes are left of the middle. Using this,and since blocks are short and each reference contains the amount ofsource bytes in the reference, it is trivial to follow exactly the rightreferences to get to the exact offset that the user wants to—and it isdone in O(log(n)). The reference stack would need to be saved in memorywhile the file handle to read from that location is active. Even withoutthe halfway marker, the order of time would still be O(log(n)) for theseek but it would take twice as long as without the halfway marker astwice as much data would need to be read to determine in which referencethe location desired to be sought to is contained in.

Modifying a File

In some embodiments, the design may perform optimally if files arewritten and then not modified again, however, it does still work thoughif files are modified—however some blocks, a logarithmic amount of thefile may need to be rewritten.

Let's say the following situation exists:

File -> B12 B9: f g h B10: d e ->B9,0,3 B12: a ->B10,2,2 c

Say the ‘g’ is to be changed to something else, say i. If B9 has onlyone reference, then it is easy as B12 must be the only one using it.Just replace it, however the block needs to get a new block sequencenumber. If it is referenced by others, then B9's reference count shouldbe reduced by 1. Then, if ‘fih’ is looked up to see if it has areference—if it does, it is noted, else ‘fih’ is added as a block anduse that in the following step (x). Then, the stack is went up to B10—itis checked to see if that has other references, if so the recount isreduced and (d, e, B->x) is added as block y, else it is changed (and anew sequence number is assigned) and the process is completed. And soon. Changing 1 block could lead to around log n other blocks also havingto change. In some embodiments, one rule that should be kept whenperforming the above operations is that one block can only referenceanother block if the sequence number of the referenced block is lower.

Reclaiming Space

Each block has a reference count and a sequence number, which only goesup, never down. The sequence number ensures that there will be nocircular dependencies. A block should only reference a block with alower sequence number. When a file is deleted, blocks need to betraversed, decreasing the reference count recursively. In someembodiments, the sequence number may go down.

However, this may be difficult to do in real time—so an implementationcould do this offline or periodically rather than doing it right away.Even if a new file references blocks which had actually reached areference count of 0 (but are still>1 in the tree), it would just add1—when the periodic check does get to the block and reduces it bywhatever amount, there will still be one left.

So what happens is that when a file is removed, only the blocks of thefile block (the block which is directly linked to refer to the file) areadded to a sorted list of blocks to reduce the reference count of andonly that one block is reduced by 1—meaning that in most cases thereference count will become 0. If any of them are already in the list,the ‘reduce count’ is just increased by 1.

When this offline space recoverer is implemented, whenever a new blockreferences a block, instead of directly increasing the count, it wouldhave to check against this table and instead decrease the reduce count.

One implementation of the algorithm could be that the order in whichthis algorithm processes the blocks to reduce the reference count ofcould do it in order of sequence number. If there is a low water markreached of items in the list, then the ones with the highest sequencenumbers are processed until the watermark is reached at which point allthe entries with the lowest sequence numbers are processed. This can bedone continuously when a writer thread is idle.

When the reference count of a block has reached 0, then the storagespace of the block can be reused. It is up to the implementation how todeal with management of free space. For example, one could have a numberof n “free block” linked lists, and a random one of these n is chosen toadd the free block to. The list could include block ranges.

Locking

If multiple writers are allowed to write to the single instanced storageat the same time, then locking should be done in a way so that the treeis only minimally affected. In fact locking should be such that readingfrom the tree should be able to continue happening while updating, forexample, the reference count or adding new items.

The way locking is performed may not affect materially the describedconcepts as locking is only one aspect of the entire picture.

However, a method to lock would be to have a fixed number of writers atany one given moment which can than be extracted out into any number ofwriters which queue their data until a writer becomes available.

If there is only one writer writing to the single instance store, thenno locking mechanism is needed, however, it has obvious advantages whenmultiple writers are allowed.

There are several types of locks. For example, Group 1: referencecounter locks, which include a read lock which still allows increasingof the reference counters by others, but no decreases; and a referencecount change lock, which allows other writers to still read the actualdata block, but not the reference count. If determined that the block isusable, the writer needs to wait for the lock to clear.

There are also Group 2: pointer locks, which include an item pointerinsert lock—put in place when a next/previous pointer needs to beupdated. For most use cases, the tree can still be scanned while this isgoing on, just not other items inserted in the same place while the lockis in place; and an item pointer delete lock—put in place when anext/previous pointer needs to be updated because an element pointing toit will be deleted. If such a lock is in place, the writer needs to waituntil the lock is released in order to go to the next/previous item.

There are also Group 3: block-level locks, which include a blocklock—this is used to change an unused block into a used block. If thislock is detected, another block should be chosen by the writer to writethe new data. These types of locks can be stored in memory or with theactual data.

Alternatives

One alternative is to do the following: Use macro blocks—128 KB (orother large block sizes)—but each macro block would be chopped up intosmall 2 kb micro blocks. These microblocks could have their hashescalculated in a rolling hash format as well as a strong hash such asSHA1 or MD5. These strong hashes could be put into a search tree sortedby the weaker rolling hashes. If there is a match, calculate the stronghash, compare it against the one in the tree and then single instanceusing the microblock in case of a match. If for a certain minimum amountof data there is no hash match, that becomes a new block. One possiblerequirement of this approach is that all 2048 bytes (or however big theblock size is) need to match. Partial block matches are not possible.

One other possible requirement of this approach is that every singlebyte needs to do a lookup in the search tree. In the primary approachdescribed herein, it is similar to following a triep data structure—thenext byte to compare against is known and only if it does not match isit necessary to start searching in the tree again—and not necessarilyfrom the root of the tree.

In the primary approach described herein, if a 1 GB file were to come upagain, it would only need to reference 1 block (though this block wouldalso increase the reference count of all blocks it refers to, but thiswould be done while scanning the search tree). In the primary approachdescribed herein, the matching blocks would be found quite quickly, asblocks will be found matching all hashes exactly.

Possible Improvements

This concepts described herein depend on being quickly able to scan asearch tree. It is possible for there to be improvements in this area.

Commercialization

Storage is growing exponentially—being able to reduce the storagerequired by both compression and single instancing at the same time isextremely valuable and saves money.

Any companies needing to store large amounts of data almost certainlystore duplicates. The concepts described herein could be used by anoperating system manufacturer creating a single instanced file system orby a backup to disk vendor. They could also be employed by databasecompanies and search companies which need to store a lot of data whichcould repeat itself to minimize storage.

A memory only implementation of this is very viable and may work verywell for storing texts/emails (especially when there is a whole emailthread—the memory used for the whole thread using this method shouldonly be only slightly larger than the last email in the thread).

Suitable Environment

FIG. 1 is a block diagram illustrating an environment 100 in whichaspects of the invention may be configured to operate. The environment100 includes one or more clients 102, a media agent 512, a singleinstance database 105 and a storage device 103. Each of the clients 102is a computing device, examples of which are described herein. Theclients 102 are connected to the media agent 512, which is also acomputing device. The media agent 512 is connected to the singleinstance database 105, which is a data store, and to the storage device103, which is also a data store. Both the single instance database 105and the storage device 103 may be any type of storage suitable forstoring data, such as Directly-Attached Storage (DAS) such as harddisks, storage devices in a Storage Area Network (SAN), e.g., a FibreChannel SAN, an iSCSI SAN or other type of SAN, Network-Attached Storage(NAS), a tape library, or any other type of storage.

The media agent 512 includes various components that perform variousfunctions. These components include a data object identificationcomponent 110, an identifier generation component 120, an identifiercomparison component 125, and a criteria evaluation component 130. Thefile identification component 110 identifies files or data objects, suchas in response to a storage operation. The identifier generationcomponent 120 generates a substantially unique identifier for the fileor data object (substantially unique identifiers are discussed in moredetail below). The identifier comparison component 125 performscomparisons of substantially unique identifiers of various files or dataobjects to determine if the files or data objects contain similar data(for example, the identifier comparison component 130 can comparesubstantially unique identifiers of two or more files or data objects todetermine if the files or data objects contain the same data; metadatasuch as access control lists (ACLs) of the two or more files or dataobjects may differ. The criteria evaluation component 130 evaluatesaspects of files or data object against a set of criteria. The mediaagent 512 may also contain other components that perform otherfunctions.

The clients 102, as part of their functioning, utilize data, whichincludes files, directories, metadata (e.g., ACLs and any other streamsassociated with the data), and other data objects. (More details as tothe storage operations involving ACLs may be found in the assignee'sU.S. patent application Ser. No. 12/058,518, entitled SYSTEM AND METHODFOR STORAGE OPERATION ACCESS SECURITY (Attorney Docket No.60692-8042.US02), the entirety of which is incorporated by referenceherein.) The data on the clients 102 is a primary copy (e.g., aproduction copy). During a copy, backup, archive or other storageoperation, the clients 102 send a copy of each data object in the datato the media agent 512. The media agent 512 generates a substantiallyunique identifier (for example, a hash value, message digest, checksum,digital fingerprint, digital signature or other sequence of bytes thatsubstantially uniquely identifies the file or data object) for each dataobject. The word “substantially” is used to modify the term “uniqueidentifier” because algorithms used to produce hash values may result incollisions, where two different files or data objects result in the samehash value. However, depending upon the algorithm or cryptographic hashfunction used, collisions should be suitably rare and thus theidentifier generated for a file or data object should be uniquethroughout the system.

After generating the substantially unique identifier for a data object,the media agent 512 determines whether it should be stored on thestorage device 103. The storage device 103 stores a secondary copy(e.g., a backup copy) of the data of the clients 102. To determine this,the media agent 512 accesses the single instance database 105 to checkif a copy or instance of the data object has already been stored on thestorage device 103. The single instance database 105 utilizes one ormore tables or other data structures to store the substantially uniqueidentifiers of the data objects that have already been stored on thestorage device 103. If a copy or instance of the data object has notalready been stored on the storage device 103, the media agent 512 sendsthe copy of the data object to the storage device 103 for storage andadds its substantially unique identifier to the single instance database105. If a copy or instance of the data object has already been stored,the media agent 512 can avoid sending another copy to the storage device103. In this case, the media agent 512 may add a reference (e.g., to anindex in the single instance database 105, such as by incrementing areference count in the index) to the already stored instance of the dataobject. As explained below, adding a reference to the already storedinstance of the data object enables storing only a single instance ofthe data object while still keeping track of other instances of the dataobject that do not need to be stored.

Instead of the clients 102 sending the data objects to the media agent512 and the media agent 512 generating the substantially uniqueidentifiers, the clients 102 can themselves generate a substantiallyunique identifier for each data object and transmit the substantiallyunique identifiers to the media agent 512 for lookup in the singleinstance database 105. If the media agent 512 determines that aninstance of a data object has not already been stored on the storagedevice 103, the media agent 512 can instruct the client 102 to send it acopy of the data object, which it then stores on the storage device 103.Alternatively, the client 102 itself can send the copy of the dataobject to the storage device 103. More details as to the generation ofthe substantially unique identifiers may be found in the assignee's U.S.patent application Ser. No. 12/058,367, entitled SYSTEM AND METHOD FORSTORING REDUNDANT INFORMATION (Attorney Docket No. 60692-8036.US05), theentirety of which is incorporated by reference herein.

The media agent 512 can support encrypted data objects. For example, oneclient 102 could generate a substantially unique identifier for a dataobject and then encrypt it using one encryption algorithm. Anotherclient 102 could generate a substantially unique identifier for anotherdata object and then encrypt it using another encryption algorithm. Ifthe two data objects are identical (meaning the two objects have thesame data, while their metadata, such as ACLs or descriptors, could bedifferent), they will both have the same substantially uniqueidentifier. The media agent 512 can then store both encrypted instancesof the data object or only a single encrypted instance. The media agent512 can also support compressed data objects. In general, the samecompression algorithm may be used to compress data objects. Therefore,the media agent 512 can generate a substantially unique identifier for adata object before or after it has been compressed. More details as tohow the media agent 512 can support encryption and compression in asingle instancing system may be found in the assignee's U.S. patentapplication Ser. No. 12/145,342, entitled APPLICATION-AWARE AND REMOTESINGLE INSTANCE DATA MANAGEMENT (Attorney Docket No. 60692-8056.US00),the entirety of which is incorporated by reference herein.

Suitable System

The above system may be incorporated within a data storage system andmay be subjected to a data stream during a data copy operation.Referring to FIG. 2, a block diagram illustrating components of a datastream 510 utilized by a suitable data storage and recovery system isshown. The data stream 510 may include client 102, a media agent 512,and a storage device 103, such as a hard disk drive, an optical diskdrive, a tape library, and/or a virtual tape library (VTL), etc. Forexample, in storage operations, the system may store, receive, and/orprepare data to be stored, copied, or backed up at a server or client.The system may then transfer the data to be stored to media agent 512,which may then refer to storage policies, schedule policies,and/retention policies (and other policies) to choose a storage device103. The media agent 512 may include or be associated with anintermediate component, to be discussed herein.

The storage device 103 receives the data from the media agent 512 andstores the data as a secondary copy, such as a backup copy. Storagedevices 103 may include magnetic tapes, optical disks, USB and othersimilar media, disk, and tape drives, and so on. Of course, the systemmay employ other configurations of data stream components not shown inFIG. 3.

Referring to FIG. 3, a block diagram illustrating an example of a datastorage and recovery system 600 is shown. Data storage systems maycontain some or all of the following components, depending on the needsof the system. FIG. 3 and the discussion herein provide a brief, generaldescription of a suitable computing environment in which the system canbe implemented.

For example, the data storage system 600 contains a storage manager 610,one or more clients 102, one or more media agents 512, and one or morestorage devices 103, 613. Storage manager 610 controls media agents 512,which may be responsible for transferring data to storage devices.Storage manager 610 includes a jobs agent 611, a management agent 612, adatabase 613, and/or an interface module 614. Storage manager 610communicates with client(s) 102. One or more clients may access datathat is stored by the system from database 622 via a data agent 621. Thesystem uses media agents 512, which contain databases 631, to transferand store data into storage devices. Client databases 622 may containdata files and other information, while media agent databases 631 maycontain indices and other data structures that assist and implement thestorage of data in storage devices 103, for example.

The data storage and recovery system may include software and/orhardware components and modules used in data storage operations. Thecomponents may be storage resources that function to copy data duringstorage operations. The components may perform other storage operations(or storage management operations) other that operations used in datastores. For example, some resources may create, store, retrieve, and/ormigrate primary or secondary data copies. Additionally, some resourcesmay create indices and other tables relied upon by the data storagesystem and other data recovery systems. The secondary copies may includesnapshot copies and associated indices, but may also include otherbackup copies such as Hierarchical Storage Management (HSM) copies,archive copies, and so on. The resources may also perform storagemanagement functions that may communicate information to higher levelcomponents, such as global management resources.

In some examples, the system performs storage operations based onstorage policies, as mentioned above. For example, a storage policyincludes a set of preferences or other criteria to be considered duringstorage operations. The storage policy may determine or define a storagelocation and/or set of preferences about how the system transfers datato the location and what processes the system performs on the databefore, during, or after the data transfer. In some cases, a storagepolicy may define a logical bucket in which to transfer, store, or copydata from a source to a data store, such as storage media. Storagepolicies may be stored in storage manager 610, or may be stored in otherresources, such as a global manager, a media agent, and so on. Furtherdetails regarding storage management and resources for storagemanagement will now be discussed.

Referring to FIG. 4, a block diagram illustrating an example ofcomponents of a server or system 700 used in data storage operations isshown. A server, such as storage manager 610, may communicate withclients 102 to determine data to be copied to storage media. Asdescribed above, the storage manager 610 may contain a jobs agent 611, amanagement agent 612, a database 613, an interface module 614, and/orother agents 720. Jobs agent 611 may manage and control the schedulingof jobs (such as copying data files) from clients 102 to media agents512 (not shown). Management agent 612 may control the overallfunctionality and processes of the data storage system or maycommunicate with global managers. Database 613 or another data structuremay store storage policies, schedule policies, retention policies, orother information, such as historical storage statistics, storage trendstatistics, and so on. Interface module 615 may interact with a userinterface, enabling the system to present information to administratorsand receive feedback or other input from the administrators or withother components of the system (such as via Application ProgrammingInterfaces (APIs)).

CONCLUSION

From the foregoing, it will be appreciated that specific examples oftechniques for single instancing may have been described herein forpurposes of illustration, but that various modifications may be madewithout deviating from the spirit and scope of the invention. As anexample, although certain types of data storage operations may have beendescribed, the system may be used to perform many types of storageoperations (e.g., copy operations, backup operations, restoreoperations, archival operations, copy operations, continuous datareplication (CDR) operations, recovery operations, migration operations,HSM operations, etc.). As another example, although it is described thatfor each byte in an input stream, it is looked up whether there existsan entry in the search tree matching the hash of the minimum sequence,this lookup may be performed for not every byte in the input stream.

More details as to single instancing may be found in one or more of thefollowing commonly-assigned U.S. patent applications: 1) U.S. Pat. App.No. 61/180,791 (entitled BLOCK-LEVEL SINGLE INSTANCING, Attorney DocketNo. 60692-8073US00); 2) U.S. patent application Ser. No. 12/145,347(entitled APPLICATION-AWARE AND REMOTE SINGLE INSTANCE DATA MANAGEMENT,Attorney Docket No. 60692-8056US00); or 3) U.S. patent application Ser.No. 12/145,342 (entitled APPLICATION-AWARE AND REMOTE SINGLE INSTANCEDATA MANAGEMENT, Attorney Docket No. 60692-8057US00), 4) U.S. patentapplication Ser. No. 11/963,623 (entitled SYSTEM AND METHOD FOR STORINGREDUNDANT INFORMATION, Attorney Docket No. 60692-8036U502); 5) U.S.patent application Ser. No. 11/950,376 (entitled SYSTEMS AND METHODS FORCREATING COPIES OF DATA SUCH AS ARCHIVE COPIES, Attorney Docket No.60692-8037US01); or 6) U.S. Pat App. No. 61/100,686 (entitled SYSTEMSAND METHODS FOR MANAGING SINGLE INSTANCING DATA, Attorney Docket No.60692-8067US00), each of which is incorporated by reference herein inits entirety. Variable instancing generally refers to storing insecondary storage one or more instances, but fewer than the total numberof instances, of each data block (or data object) in a set of data(e.g., primary data). More details as to variable instancing may befound in the commonly-assigned U.S. Pat. App. No. 61/164,803 (entitledSTORING A VARIABLE NUMBER OF INSTANCES OF DATA OBJECTS, Attorney DocketNo. 60692-8068US00).

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof means any connection or coupling,either direct or indirect, between two or more elements; the coupling orconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import, when used in this application, refer tothis application as a whole and not to any particular portions of thisapplication. Where the context permits, words in the above DetailedDescription using the singular or plural number may also include theplural or singular number respectively. The word “or,” in reference to alist of two or more items, covers all of the following interpretationsof the word: any of the items in the list, all of the items in the list,and any combination of the items in the list.

The above Detailed Description of examples of the invention is notintended to be exhaustive or to limit the invention to the precise formdisclosed above. While specific examples for the invention are describedabove for illustrative purposes, various equivalent modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize. For example, while processes or blocks arepresented in a given order, alternative implementations may performroutines having steps, or employ systems having blocks, in a differentorder, and some processes or blocks may be deleted, moved, added,subdivided, combined, and/or modified to provide alternative orsubcombinations. Each of these processes or blocks may be implemented ina variety of different ways. Also, while processes or blocks are attimes shown as being performed in series, these processes or blocks mayinstead be performed or implemented in parallel, or may be performed atdifferent times. Further any specific numbers noted herein are onlyexamples: alternative implementations may employ differing values orranges.

The teachings of the invention provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various examples described above can be combined to providefurther implementations of the invention.

Any patents and applications and other references noted above, includingany that may be listed in accompanying filing papers, are incorporatedherein by reference. Aspects of the invention can be modified, ifnecessary, to employ the systems, functions, and concepts of the variousreferences described above to provide yet further implementations of theinvention.

These and other changes can be made to the invention in light of theabove Detailed Description. While the above description details certainembodiments of the invention and describes the best mode contemplated,no matter how detailed the above appears in text, the invention can bepracticed in many ways. Details of the system may vary considerably inimplementation details, while still being encompassed by the inventiondisclosed herein. As noted above, particular terminology used whendescribing certain features or aspects of the invention should not betaken to imply that the terminology is being redefined herein to berestricted to any specific characteristics, features, or aspects of theinvention with which that terminology is associated. In general, theterms used in the following claims should not be construed to limit theinvention to the specific embodiments disclosed in the specification,unless the above Detailed Description section explicitly defines suchterms. Accordingly, the actual scope of the invention encompasses notonly the disclosed embodiments, but also all equivalent ways ofpracticing or implementing the invention under the claims.

While certain aspects of the invention are presented below in certainclaim forms, the applicant contemplates the various aspects of theinvention in any number of claim forms. For example, if one aspect ofthe invention is recited as a means-plus-function claim under 35 U.S.C.§112, sixth paragraph, other aspects may likewise be embodied as ameans-plus-function claim, or in other forms, such as being embodied ina computer-readable medium. (Any claims intended to be treated under 35U.S.C. §112, ¶ 6 will begin with the words “means for”, but use of theterm “for” in any other context is not intended to invoke treatmentunder 35 U.S.C. §112, ¶ 6.) Accordingly, the applicant reserves theright to add additional claims after filing the application to pursuesuch additional claim forms for other aspects of the invention.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thespirit and scope of the invention. Accordingly, the invention is notlimited except as by the appended claims.

1. A method of deduplicating data, wherein the method is performed by acomputing system having a processor and memory, the method comprising:storing multiple blocks of data in a data structure, wherein: themultiple blocks of data have an first size, the multiple blocks of datarepresent a set of data having a second size that is greater than thefirst size, a block of data is associated with multiple firstidentifiers, and the multiple blocks of data are sorted in the datastructure by the associated multiple first identifiers; receiving afile, the file including multiple bytes; accessing the data structure;based at least partly upon accessing the data structure, determining, bythe computing system, whether one or more of the multiple bytes arealready stored, wherein the number of the one or more bytes is less thana number of bytes in a block of data; and storing bytes that are notalready stored using a storage device.
 2. The method of claim 1, furthercomprising generating multiple second identifiers for the file, whereinthe multiple second identifiers for the file include powers of 2 rollinghashes.
 3. The method of claim 1, further comprising generating multiplesecond identifiers for the file, wherein the multiple second identifiersinclude powers of 2 rolling hashes, and wherein determining whether theone or more of the multiple bytes are already stored includes comparingthe multiple second identifiers for the file with the multiple firstidentifiers associated with the multiple blocks of data.
 4. The methodof claim 1 wherein the data structure includes a hierarchical datastructure that includes multiple nodes, and wherein a first node canreference data in any other node excepting nodes that are descendants ofthe first node.
 5. The method of claim 1 wherein multiple blocks of dataare compressed.
 6. A computing system for deduplicating data, thecomputing system comprising: a processor; a memory; means for receivinga first set of data; means for dividing the first set of data into atleast one block, the block including a total number of bytes; means foraccessing a searchable data structure, the searchable data structureincluding a second set of data; means for determining whether one ormore bytes of the block are included in a portion of the second set ofdata in the searchable data structure, wherein the number of the one ormore bytes is less than the total number of bytes of the block; meansfor replacing the one or more bytes with a reference to the portion ofthe second set of data, wherein the means for replacing replaces the oneor more bytes with a reference to the portion of the second set of dataif the one or more bytes of the block are included in the portion of thesecond set of data in the searchable data structure; and means forstoring the block.
 7. The computing system of claim 6, furthercomprising means for generating powers of 2 rolling hashes.
 8. Thecomputing system of claim 6 wherein the searchable data structureincludes a hierarchical data structure that includes multiple nodes, andwherein a first node can reference data in any other node exceptingnodes that are descendants of the first node.
 9. The computing system ofclaim 6, further comprising means for compressing data.
 10. A system fordeduplicating data in a data storage network, wherein the data storagenetwork includes one or more storage devices coupled via a network, andwherein the network also couples to one or more computing systems, thesystem comprising: one or more storage devices storing a searchable datastructure, the searchable data structure including a first set of data;and a computing system configured to: receive a second set of data;divide the set of data into at least one block, the block including atotal number of bytes; access the searchable data structure; determinewhether one or more bytes of the block are included in a portion of thefirst set of data in the searchable data structure, wherein the numberof the one or more bytes is less than the total number of bytes of theblock; replace the one or more bytes with a reference to the portion ofthe second set of data if the one or more bytes of the block areincluded in the portion of the first set of data in the searchable datastructure; and store the block using the one or more storage devices.11. The system of claim 10 wherein the computing system is furtherconfigured to generate powers of 2 rolling hashes for the second set ofdata.
 12. The system of claim 10 wherein the searchable data structureincludes a hierarchical data structure that includes multiple nodes, andwherein a first node can reference data in any other node exceptingnodes that are descendants of the first node.
 13. The system of claim 10wherein the computing system is further configured to compress thesecond set of data.